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Abstract — We consider the Linear Programming (LP) solu- 
tion of the Compressed Sensing (CS) problem over reals, also 
known as the Basis Pursuit (BasP) algorithm. The BasP allows 
interpretation as a channel-coding problem, and it guarantees 
error-free reconstruction with a properly chosen measurement 
matrix and sufficiently sparse error vectors. In this manuscript, 
we examine how the BasP performs on a given measurement 
matrix and develop an algorithm to discover the sparsest vectors 
for which the BasP fails. The resulting algorithm is a generaliza- 
tion of our previous results on finding the most probable error- 
patterns degrading performance of a finite size Low-Density 
Parity-Check (LDPC) code in the error-floor regime. The BasP 
fails when its output is different from the actual error-pattern. 
We design a CS-Instanton Search Algorithm (ISA) generating a 
sparse vector, called a CS-instanton, such that the BasP fails on 
the CS-instanton, while the BasP recovery is successful for any 
modification of the CS-instanton replacing a nonzero element 
by zero. We also prove that, given a sufficiently dense random 
input for the error-vector, the CS-ISA converges to an instanton 
in a small finite number of steps. The performance of the CS- 
ISA is illustrated on a randomly generated 120 x 512 matrix. 
For this example, the CS-ISA outputs the shortest instanton 
(error vector) pattern of length 11. 



I. Introduction 



A. Background 



Compressed sensing (CS) [1] is a computational technique 
to recover a sparse signal from a small set of measure- 
ments. Given the measurements, described in terms of the 
so-called measurement matrix, the ideal CS looks for the 
sparsest signal, i.e. minimizing the ^o-norm, consistent with 
the measurements. This ideal formulation is known to be 
NP-hard [2], [3]. A key practical observation of the CS 
theory came with the Basis Pursuit (BasP) algorithm of 
Chen, Donoho and Sanders [4] who suggested relaxing 
the difficult ^o-norm minimization to ^i-norm minimization, 
which is convex and thus computationally tractable. The 
BasP heuristics have shown remarkable performance, that 
was theoretically explained in the breakthrough paper of 
Candes and Tao [5], stating the CS problem as a linear 
channel coding problem involving recovery of an input real 
vector from its corrupted image. In [5], it was proved that the 
£i relaxation guarantees perfect recovery of the input vector 
for sufficiently sparse error vectors and a properly chosen 
measurement matrix. The conditions on the measurement 
matrix were expressed in terms of the so-called Restricted 
Isometry Property (RIP). It was also shown in [5] that a 



measurement matrix drawn from a proper random ensemble 
possesses the RIP with high probability. 

Estimation of the fraction of tolerable errors for a random 
measurement matrix became an important follow up ques- 
tion. However, extending the approach of [5] to a given finite 
measurement matrix is not easy, as checking if the matrix has 
the RIP becomes computationally hard. This aspect of the CS 
is reminiscent of similar problem in the so-called expander 
graph based analysis [6] of the Low-Density Parity-Check 
(LDPC) codes [7]. In fact, expander-based constructions of 
CS measurement matrices have been investigated in [8], [9]. 

The relation between LDPC codes and CS was further 
explored in the work of Dimakis and Vontobel [10], who 
juxtaposed two linear programs: the BasP of CS which can 
be restated as a Linear Program (LP) and the so-called LP- 
decoder [11] of a related LDPC code. It was shown in [10] 
that a binary matrix which is a good parity check matrix 
for LP decoding of the corresponding LDPC code is also a 
good zero-one measurement matrix over reals for BasP in 
the respective CS problem. The authors of [10] also pointed 
out other directions in which the relationship between BasP 
and LP-LDPC can be investigated to translate theoretical 
guarantees from one field to the other This challenge, of 
extending descriptive results from the LDPC world into the 
world of CS reconstruction, has motivated this work. 

B. Overview of Results 

In this paper, we pose the following question: Given a 
measurement matrix good for the ti-norm recovery, how 
does one find the sparsest vectors for which the li-norm 
minimization fails? A similar question arises in decoding of 
a finite LDPC code by Belief Propagation [12] or other sub- 
optimal decoding algorithms, notably LP decoding, in the so- 
called error-floor regime [13], [14], [15]. Roughly speaking, 
the error floor is an abrupt degradation of the frame error rate 
performance of a code in the high signal-to-noise ratio (SNR) 
regime caused by low-weight error patterns not correctable 
by the sub-optimal decoder. Algorithms to enumerate such 
rare noise configurations (known as instantons, i.e. instances 
in the space of noise configurations) are crucial to the design 
of better decoders as well as codes. Similar statements can 
be made in the CS setting, where identifying sparsest error 
vectors for which the BasP fails is important to assess the 
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Fig. 1. Fig. (a) shows example of a (15 X 64) measurement matrix, F, with 
orthonormal rows. Fig. (b) shows a sequence of en'or vectors (e'"' — e'^') 
representing the ISA for the measurement matrix from (a). The first vector 
(with 15 nonzero elements) was selected at random. It takes four iterations 
of the ISA (four columns in the Fig. (b)) to reach an instanton, i.e. the 
error-pattern for which BasP decoding fails, containing only three nonzero 
elements. 



quality of a given measurement matrix as well as to guide 
the design of better measurement matrices. 

The main contribution of this paper is the formulation of 
the CS instanton search algorithm (CS-ISA). Our approach 
to the problem of identifying instantons for CS consists of 
extending and adapting the ISA developed originally in [16], 
[17], [18] for analysis of the error-floor of a given LDPC 
code, to the problem of the worst configuration recovery in 
CS. Given a measurement matrix (e.g. the 15 x 64 matrix 
shown in Fig. |l(a)[ ), the algorithm starts with a dense error 
vector for which the BasP fails and iteratively finds error 
vectors with a smaller Iq norm that result in BasP failure. 
At some point, the algorithm finds an instanton, defined as 
a sparse error vector not correctable by BasP such that any 
of its further reductions in the £o norm is BasP correctable. 
A typical sequence of CS-ISA, described in Section |III1 is 
illustrated in Fig. |l(b)| 

Initializing ISA with an error vector at random allows us to 
sample the space of instantons for a given measurement ma- 
trix efficiently. We describe the quality of the measurement 



matrix in terms of the distribution of instantons over their 
sparsity. (See the bar-diagram of Fig. |2] for an illustration.) 
This distribution is the practical tool that we propose to use 
as a guide for constructing better measurement matrixes. 

Finally, we show that the CS-ISA comes with some 
theoretical guarantees. We prove in Section |V] that for any 
sufficiently noisy initialization, the CS-ISA converges in a 
finite number of steps, which is significantly smaller than the 
measurement matrix size. (Each step of the CS-ISA requires 
running a single instance of the BasP algorithm.) 

After the submission of the initial version of this 
manuscript, we have been made aware of related approach 
by Dossal, Peyre and Fadili [19], who proposed a greedy 
pursuit algorithm that computes, for a given measurement 
matrix, sparse vectors for which the BasP fails. 

The rest of the manuscript is organized as follows. Section 
Hn introduces the problem setting and related terminology. 
Section [III] describes the CS-ISA. Section |IV] illustrates 
the performance of the CS-ISA in sampling the space of 
instantons of a randomly generated (120 x 512) measurement 
matrix. We present a brief sketch of the theorem, proving 
the convergence of the algorithm in a small finite number of 
steps in Section [V] Section [Vl] is reserved for discussions. 

II. Compressed Sensing Preliminaries; Problem 
Setting 

In this Section, we discuss the BasP algorithm adopting 
formulation and terminology from [4], [5]. (The interested 
reader is referred to [20] for a comprehensive list of ref- 
erences on CS.) The problem setting is as follows. An 
original real-valued information vector / S M" is trans- 
formed (coded) into a longer vector Af G M™, where A S 
jgmxn ^ fi^jj matrix known as the generator matrix. 
The result, transmitted over the CS-channel, is received 
as y = Af + e, where e G M™ is the unknown error 
vector, assumed to be sparse. Recovering the error vector 
e is sufficient to reconstruct the information vector /, as 
knowledge of y along with e gives Af and hence / can be 
recovered straightforwardly as A is full rank. Now, consider 
a p X m matrix F such that FA ~ 0. F will be called the 
measurement matrix. It follows that y ~ Fy = Fe and the 
problem of recovering the sparse error vector e is equivalent 
to the problem of finding the sparsest vector d subject to 
Fd = y. The CS-decoding/reconstruction is successful if 
d — e. The sparsest solution to this problem can be found 
by solving the following optimization problem (CS-OPT) [5] 



mm 



(1) 

Fd=Fy 

where the £q norm of a vector d measures its sparsity, i.e. 
the number of nonzero entries, \\d\\ig = \{i : di ^ 0}|. 

As stated in Eq. ([B the problem is NP hard [2], [3] (of 
exponential complexity in m) and [4] suggested a relaxed 
(weaker but tractable) version of the CS-decoding, coined 
Basis Pursuit (BasP): 



min lldll 



(2) 



Fd=y=Fe 



where ^o-norm is replaced by the ^i-norm, \\d\\e-^ = 
X]"=i Mil- ^1- ® ^Iso tie recast as an LP. 

As shown in [5], the BasP is capable of exact reconstruc- 
tion, i.e. d = e, under the conditions that (1) the measurement 
matrix has RIP, and (2) e is sufficiently sparse. Formally, 
[5] states that if the RIP constants 6s , 6s and 6s,2S satisfy 
5s + Os + 9s.2S < 1, then for any error vector e with 
II e 1 1^0 < S, e is the unique solution of both Eq. ([T]l and 
Eq. (|2]l. However, finding the maximum value of S for which 
the condition Ss + Os + Ss,2S < 1 holds is, in general, a 
difficult problem and the most recent state-of-the-art results 
provide only an estimate for S/ra in the asymptotic limit 
of large samples [5]. Moreover, these estimates for the RIP- 
constants are normally very loose and cannot be used to 
evaluate the quality of reconstruction of practically important 
small-to-moderate sized matrices. 

On the other hand, brute force search techniques for find- 
ing sparse vectors that lead to BasP failures are prohibitively 
expensive. In fact, BasP with a well-tuned measurement 
matrix performs on typical instances of the error-vector 
really well. Thus, using standard (Monte Carlo) sampling 
techniques will typically not deliver a failure. Hence, there 
is a need to develop smart techniques sampling the space of 
failures (called instantons) of the given measurement matrix. 

III. INSTANTON SEARCH ALGORITHM FOR BASP 

In this Section, we provide a formal description of the CS- 
ISA. We say that the BasP fails on a vector e\f e ^ d, where 
d solves Eq. (|2]i. We start with the following two definitions. 

Definition 1 (Instanton): Let e be a fc-sparse vector (i.e. 
the number of nonzero entries in e is equal to k). Consider 
an error-vector e', derived from e by replacing one of its 
nonzero component by zero (thus e' is {k — l)-sparse). e is 
an instanton if the BasP fails on e while it succeeds on any 
e' derived from e. 

Definition 2 (Median): Let e be a vector and let t denote 
the smallest number such that the sum of the t largest entries 
(in absolute value) of e is at least equal to ||e||£j/2. Let 
T = {ii, Z2, . . . , it} denote indices of the t largest entries of 
e. Then, the median of e is the vector e with support T and 
Ci = Ci for i Cz T. 

Fact 1: If e is a vector such that Fe = 0, then BasP 
fails on e. To see this, observe that ||e — e||fj < \\e\\e-^ and 
\\e-e\\e„ > \\e\\i„. 

Now we are ready to describe the Instanton Search 
Algorithm: 

• Initialization (/ = 0) step: Initialize the algorithm to a 
vector e'^'^' of length m with a sufficient number of 
errors such that BasP fails on e^"'', i.e. applied to e*^*^-* 
BasP produces another vector e^'^-' 7^ e^'''. 

• ^ > 1 step: Consider the vector e^'^^' = e^^^^'> —eS^^^\ 
where e^'^^' denotes the output of BasP acting on 
e^'^^-*. Let e*-'"^^ denote the median of e^'^^'. Only 
two cases arise (see Lemma [U: 

(i) If ||e('-i)||f„ < ||e('-i)||^„, then e^') = e^'-^) is the 



l-th step output/(Z + l)-th step input. (Note that by Fact 
1, BasP fails on e^')) 

(ii) If ||e('-i)||^o = ||e('-i)||f„, define L = 
{ii, 12, ... , ik, } as the support of e^'^^^ Let Lj^ = L\it 
for some it € L. Let r'* be a vector such that rj = 

e^l~'^\j G Li^ and contains zero elements elsewhere. 
Apply BasP to all r'* and denote the it output by f**. If 
pt _ j.it fQj. ^jj j-jjgjj g('-i) is the desired instanton 

and the algorithm halts. Else, e*-'' is set to r*' for some 
it for which ^ r**. 

Starting with some random non-sparse initialization of the 
error vector for which BasP fails, the algorithm aims to get 
iteratively as close as possible to the zero error vector while 
keeping the failed status for BasP when it acts on the current 
vector. In the ISA (which should in fact, for the sake of 
accuracy, be called ISA for the Zo-norm channel), this aim 
is achieved by alternating the BasP- and the median- steps. 
The logic behind the alternation is obvious: when BasP fails 
we apply the median-step to reduce the size of e, defined as 
the difference between the original e and e. 

We will prove in Section |V] that the ISA algorithm defined 
above outputs an instanton in a small number of steps. How- 
ever, prior to that we illustrate the algorithm performance on 
an example in the next Section. 

IV. Example of the Error-Surface Exploration 

In this Section, we first consider a 15 x 64 measurement 
matrix, used to show the performance of the ISA in Fig. [T] 
This toy example, described in details in the next paragraph, 
while not useful in practice, allows a reasonable visualization 
and thus serves us mainly for illustration purposes. After- 
wards, and focused on a tour-de-force demonstration, we 
discuss a more realistic 120 x 512 example. 

The 15 X 64 measurement matrix, shown in Fig. |l(a)[ 
is built by first creating a 15 x 64 matrix H with i.i.d. 
Gaussian entries and then finding the orthonormal basis of 
the space spanned by the columns of H. The dynamics 
of the ISA is illustrated in Fig. |l(b)| The error vector 
initialization is random and it happens to be with support 
of size 15 for the illustrative example. For this example 
locations of the nonzero components of the initialization 
are, [1 5 8 12 19 31 34 35 37 43 45 51 57 59 62], 
and their respective values are, 

[1.1738 0.6554 - 2.2990 - 0.1783 - 1.1907 - 0.4254 
- 0.4768 2.0385 - 0.0695 0.6997 0.4137 2.9185 0.6545 
1.1149 0.6789]. BasP fails on this error pattern producing the 
output e°. In step 1, we compute the median of e° resulting 
in a vector with support [1 19 35 51 56 62] and components 
[1.1738 - 1.1907 1.1590 1.3883 1.0888 0.6789]. Next, 
we set e^^-* = e*^*^'. In step 2, we compute the median 
of e^^-' resulting in a vector with support [1 19 51 56] 
and components [1.1738 - 1.1907 1.0843 1.0888]. 
Hence, we set e^^^ = 6*-^^ In step 3, we compute e*^^) 
yielding the vector with support [1 19 56] and components 
[1.1738 - 1.1907 1.0888], thus resulting in e^^) = e^^). 
In step 4, the median of e*^"^^ generates vector with support 
[1 19 56] and components [1.1738 - 1.1907 1.0138]. 




Fig. 2. Bai'-diagram showing number of instance per recorded instanton 
length found in the result of 5000 trial runs of the ISA on a random 
measurement matrix of size (120 X 512) and with random initializations 
of the sparsity (^o-norm) 40. 
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derived from 
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we consider all the error vectors 



p(3) 



by replacing one (of the three) nonzero 



components by zero. BasP applied to these newly derived 



vectors decodes correctly, and thus 
instanton of the measurement matrix. 



s'^^) is declared an 



In order to illustrate the effectiveness of the ISA in 
sampling the space of instantons, we consider a random 
sample of (120 x 512) measurement matrix with orthogonal 
rows. We run the ISA 5000 times, generating initialization 
for the error-vector always with 40 initial errors and record 
the length of the instanton found for each trial. (40 is a suf- 
ficiently large integer chosen to guarantee that the majority 
of random error-vector initializations leads to BasP failure 
and to an instanton configuration consequently, while only a 
small portion of initializations of the £o-norm 40 are decoded 
correctly by BasP, and these rare and not interesting instances 
are simply ignored. Note that lowering the initialization 
sparsity will lead to wasting a lot of initializations as these 
would be typically decoded by BasP without an error.) The 
resulting distribution of the instanton lengths is shown in 
Fig. ID The smallest length instanton discovered by the ISA 
for this example is 11. Note that it is still possible that there 
exist instantons of length less than 11. 

The ISA was implemented in MATLAB and the BasP step 
in the algorithm was solved using the "11 -magic" package 
from [21]. For the aforementioned measurement matrix of 
size (120 X 512) and the initial error-vector of the £o-norm 
40, the average time of the BasP and the ISA runs were 0.2 
seconds and 4.2 seconds respectively on a (laptop) Intel core 
duo 2GHz processor 

V. Proof of the CS-IS Algorithm Correctness 

In this Section, we estabUsh correctness of the CS-ISA and 
prove that the ISA outputs an instanton in a finite number 
of steps. 



Lemma 1: Let e be a vector and let e denote the output 
of BasP on e. Further, let e = e — e and denote the median 
of e by e. Then, 

PIko < l|e||fo 

Proof: Let 5*1 denote the support of e and J denote 
the support of e. Since e is the BasP output for e, we have 



> e 



Or stating it differently. 



E 



^ E 



(3) 
(4) 



\e^\) > 2^ |e,| 

ieSi j€J\Si 

Now consider, X^ieSi l^'l- ha\e 

E i^'i = E 1^* ~ - E*^!^*! ~ - E i^ji 

ieSi ieSi ieSi j&J\Si 

= E I^J"^jl= E i^ji 

Eq. (|5]l implies that the sum of entries of e in Si is at least 
equal to the sum of entries of e over the remaining entries. 
This implies that the £o norm of the median of e cannot 
exceed \Si\ = ||e||<>„. ■ 

Corollary 1: The ISA converges to an instanton in a finite 
number of steps. 

Proof: At every step I of the ISA, we have ||e*^'^||fo < 
Ik''"^"* Iko- construction, the output of the ISA is an 
instanton. ■ 

VI. Discussion and Future Work 

We envision further development of the Instanton Search 
technique along the following lines. 

• Sampling of the instanton space provided by the ISA is not 
uniform. An important future task is to design a uniform- 
sampling modification of the ISA. 

• Studying how ISA complexity scales with the measure- 
ment matrix size constitutes another important problem. ISA 
outputs an instanton in a linear (in the number of columns 
of measurement matrix) number of steps in the worst case. 
However, it is still not clear how many instances of the ISA 
should be executed till the smallest instanton is found. While 
the answer clearly depends on the particular measurement 
matrix, to answer this question for an average case would 
also be of interest. 

• If the error-vector is almost sparse, or alternatively if the 
measurements are noisy, the problem of exact CS recon- 
struction is replaced by an approximate reconstruction. One 
possible modification of the BasP algorithm, the so-called 
Lasso algorithm [22], consists in adding to the ^i-norm of 
the signal (the original BasP) a part linear in the ^2-norm 
of the noise. With a proper (soft) definition of failure, one 
should be able to extend the ISA approach to finding the 
worst (highest probability) configurations leading to Lasso's 
failures. 



• The ISA technique allows adaptation to the problem of 
Matrix Completion via the computationally tractable min- 
imization of the matrix nuclear norm (replacing exact but 
not tractable minimization of the matrix rank) [23], [24]. In 
this setting, a sparse and properly conditioned matrix (which 
one aims to reconstruct) is fixed and the question becomes 
exploring the set of measurements required for the nuclear- 
norm minimization to succeed. The desired modification of 
the ISA should be capable of sampling the most dangerous 
configurations of measurement, for example defined as a 
set of measurements such that their nuclear-norm based 
decoding leads to a failure, while addition of any single non- 
zero measurements results in a successful decoding. 

• We can also adapt the ISA algorithm to the sequential 
compressed sensing formulation of Maloutov et al [25]. In 
fact, the modification is rather straightforward as it only 
replaces BasP, as a sub-step of the ISA, by its sequen- 
tial implementation (also imitating the order in which the 
measurements are received). A similar modification can be 
implemented for any other LP-based modification of BasP, 
such as the Subspace Pursuit algorithm discussed in [26]. 
Note that this streamlining of BasP is somehow similar to 
the adaptive realization of the LP decoding of LDPC codes 
suggested in [27]. 

• However successful the BasP algorithm is, it is still 
suboptimal with respect to the exact ^o-norm minimization. 
It will thus be important to study the gap between exact and 
suboptimal decodings, in particular constructing a sequence 
of convex optimizations improving the performance of BasP 
gradually. This sequence may be designed along the lines 
of discussion in [10], illustrated on example of the — 1 
measurement matrix and error-vector that the LP-LDPC is 
stronger than the respective BasP. The ISA-approach can be 
easily adapted to such convex improvements over BasP. 

• We envision that the most interesting (but also the most 
challenging) application of this Instanton Search approach 
will be in designing a good measurement matrix, in the spirit 
of how one may think about using the instanton search for 
selecting a good LDPC-parity check matrix (say picked from 
given random ensemble optimized with respect to its water- 
fall behavior) with the lowest error-floor. 
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